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whether or how a given algorithmic behavior should be addressed. 

If a representation Is factually accurate, can It still be algorithmic unfairness? 

Yes. For example, imagine that a Google image query for "CEOs" shows predominantly men Even if 
it were a factually accurate representation of the world, it would be algorithmic unfairness because it 
would reinforce a stereotype about the role of women in leadership positions. However, factual 
accuracy may affect product policy's position on whether or how it should be addressed In some 
cases, it may be appropriate to take no action if the system accurately affects current reality, while in 
other cases it may be desirable to consider how we might help society reach a more fair and 
equitable state, via either product intervention or broader corporate social responsibility efforts 

If a system's behavior Is not intended, can It still be algorithmic unfairness? 

Yes. If the behavior is unfair, it meets the definition regardless of the root cause 


Definition 

"algorithmic unfairness" means unjust or prejudicial treatment of people that is related to sensitive 
characteristics such as race, income, sexual orientation, or gender , 5 through algorithmic systems or 
algorithmically aided decision-making. 
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SOURCE QUALITY 


Revamping News Corpus 

Goal: Establish “single point of truth’ for definition of “news’ across Google products. Mitigate risk of 
low-quality sources and misinformation in Google News corpus. 

Status: Define new Google News corpus utilizing existing infrastructure and tools while integrating 
different quality tiers and labels, algo-human content review, new cross-product review/exclusion 
pipeline and updated inclusion Ul. 

CONTACTS: 

MORE INFO: Proposal, Updated rater template (WIP) / Stress Test publisher list 

NEXT STEPS: Research and gather info about possible signals to leverage for product teams to make 

source quality decisions. Finalize and test new rater inclusion guidelines with standards to combat 

misinformation. 
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Definition of Algorithmic Unfairness 



Last Updated 

February 2017 

go/algorithmic-unfairness-definition 

Goals 

Our goal is to create a company-wide definition of algorithmic unfairness that: 

1. Articulates the full range of algorithmic unfairness that can occur in products. This 
definition should be robust across products and organizational functions. 

2. Establishes a shared understanding of algorithmic unfairness for use in the development of 
measurement tools, product policy, incident response, and other internal functions. 1 

3. Is broadly consistent with external usage of the concept. While it is not a goal at this time to 
release this definition externally, it should represent external concerns so that we can ensure 
our internal functions address these concerns. 23 3 - 4 

Non-Goals 

The following are not goals for this document: 

1. Specify whether and how Google will take action on potential instances of unfairness that 
involve the use of an algorithm. This will fall instead to product policy. 

2. Describe the consequences of algorithmic unfairness and why they matter. 

Definition 


"algorithmic unfairness" means unjust or prejudicial treatment of people that is related to sensitive 
characteristics such as race, income, sexual orientation, or gender, 5 through algorithmic systems or 
algorithmically aided decision-making. 


' The discussion in Lipton (2016) suggests that instances that present as identical product behavior may upon investigation be 
revealed to have substantially different root causes and therefore require different remediations. Because the nature of an instance 
may not be evident a priori, a wide range of instances are likely to be reported and investigated together. (Zachary Chase Lipton. The 
Deception of Supervised Learning . KDnuggets News 16:n33, September 2016.) 

2 Peter Swire. Lessons from Fair Lending Law for Fair Marketing and Big Data . Future of Privacy Forum, and presented before the 
Federal Trade Commission Workshop on "Big Data: A Tool for Inclusion or Exclusion?" (2014). 

3 Solon Barocas and Andrew D. Selbst. Big Data's Disparate Impact . 104 California Law Review 671 (2016). 

4 ACM US Public Policy Council. Statement on Algorithmic Transparency and Accountability . January 12, 2017. 

5 This list is not intended to be exhaustive, and additional examples of sensitive characteristics appear in the FAQ. 
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Scope 

The following are in scope for algorithmic unfairness: 

In algorithms that drive predictive systems (e g., personalization or device financing), this definition 
encompasses automated actions significantly adverse to the interests of a user or group of users, 6 
on the basis of a characteristic that is sensitive in the context of a particular interaction. 

In algorithms that drive representational systems (e.g., search), this definition encompasses the 
implied endorsement of content likely to shock, offend, or upset users sharing a sensitive 
characteristic, or to reinforce social biases. 7 

The following are not in scope for this definition but may be covered by other internal definitions and 
policies: 

1. Biased content, for example user-generated content that appears in products (while the 
content itself is not in scope, the algorithmic handling of such content may be in scope) 

2. Insensitive designs which occasion unfairness, and/or interface designs that are less usable 
by groups with sensitive characteristics 

3. Google's internal decisions such as hiring or compensation, which may be influenced by 
algorithms 

Test Cases for Scope 

The following are illustrative examples of what is (and is not) in scope for algorithmic unfairness. Of 
the examples that are in scope, a subset may be determined by product policy to require 
remediation. 

Google Display Ads for High-Paying Jobs 

CMU published a study in 2015 with experiment-based observations, arguing that Google's ad 
serving system perpetuates gender bias on the basis of two campaigns that were found to target 
high salary jobs at male users on the Times of India website. 8 The effect was highly sensitive to the 
ads from this particular service, and the same effect was not reproduced in several other 
experiments by the same authors. The cause was found to be higher CPA (cost per conversion) for 


6 This includes effects which are small for a single instance but have a significant cumulative effect. As Greenwald et al. observe, 
statistically small effects can have substantial societal impact when they apply to many people, or if they apply repeatedly to the 
same person. (Anthony G. Greenwald, Mahzarin R. Banaji, and Brian A. Nosek. Statistically small effects of the Implicit Association 
Test can have societallv large effects . Journal of Personality and Social Psychology, 108:553-561, 2015.) 

7 Note that even small reinforcement biases in systems can be magnified via positive feedback loops, since biased representations 
can influence human behavior which is in turn fed back into training data for those systems. 

8 Amit Datta, Michael Carl Tschantz, and Anupam Datta. Automated Experiments on Ad Privacy Settings . PETS 2015, pp. 92-112, 
June 2015. 
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female users for one campaign (notably with a higher CTR for female users) and advertiser targeting 
to male-only users for the other. 9 

In scope for algorithmic unfairness. In the first case, bias in user behavior trained the system to have 
biased targeting. In the second case, advertiser-selected criteria were related to a sensitive 
characteristic. 

Facebook Computation of Unregulated FICO Scores 

Researchers raised concerns in 2015 about Facebook computing non-regulated credit scores based 
on user activity. 10 One substantial issue they commented on is that credit scores are regulated by 
the Equal Credit Opportunity Act of 1974, which prohibits creditors from discriminating against 
applicants on the basis of race, religion, national origin, sex, marital status, age, or receiving public 
assistance; however, the non-regulated scores did not appear to have such restrictions. * 11 

In scope for algorithmic unfairness. Based on the user's behavior, the system automatically 
computed sensitive characteristics that could adversely affect their financial opportunities. 

Chumhum Suppression of Businesses in Fligh-Crime Areas 

An episode of the primetime television drama "The Good Wife" featured a tortious interference case 
against a fictional search engine company (Chumhum) for releasing a maps application that 
suppressed businesses in high crime areas. 12 

In scope for algorithmic unfairness. Content was excluded by an algorithm in a way that 
disproportionately affected people who owned businesses in neighborhoods correlated with the 
sensitive characteristics race and income. 


West African Spam Filters 

An external researcher poses the following: "One question is whether the design of spam filters 
could make certain individuals more susceptible to having their legitimate messages diverted to 
spam folders. For example, does being located in a hotbed of Internet fraud or spam activity, say 
West Africa (Nigeria or Ghana) or Eastern Europe, create a tendency for one's messages to be 
mislabeled as spam?" 13 

In scope for algorithmic unfairness. In this hypothetical, the system learns associations and 
downgrades or excludes content related to sensitive characteristics such as national origin and 
race. 


9 Giles Hogben, Alex McPhillips, Vinay Goel, and Allison Woodruff. Allegations of Algorithmic Bias: Investigation and Meta-Analysis . 
September 2016. oo/alleoations-of-algorithmic-bias 

10 Tressie McMillan Cottom. Credit Scores. Life Chances, and Algorithms . May 30,2015. 

11 Astra Taylor and Jathan Sadowski. How Companies Turn Your Facebook Activity Into a Credit Score: Welcome to the Wild West 
of Data Collection Without Regulation . The Nation, May 27, 2015. 

12 The Good Wife, Episode 'Discovery'. CBS, first aired November 22,2015. 

13 Jenna Burrell. How the machine "thinks': Understanding opacity in machine learning algorithms . Big Data & Society 3(1 ):1 -12, 
2016. 
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Google Photos Use of Gorillas Label 

In June 2015, a web developer posted on Twitter that Google Photos had tagged an image showing 
him and a friend at a concert with the label "Gorillas". "Of all terms, of all the derogatory terms to 
use," Alcine said later, "that one came up.” According to a post mortem, Google executives noticed 
Alcine's tweets within one hour. A Googler reached out through Twitter for permission to access the 
user's photos, and the issue was identified and resolved. In the short term, the Photos team stopped 
suggesting the "Gorillas” tile in the Explore page and stopped showing Search results for queries 
relevant to gorillas; in the long term, the team pledged to investigate the image annotation models 
that generate false positives for the gorilla label and work on the image annotation pipeline and 
quality evaluation processes. 14 

In scope for algorithmic unfairness. The system drew an incorrect inference that is offensive to 
members of a given race. 

Autocomplete Results for Trayvon Martin 

In 2013, Autocomplete results showed negative results for Travyon Martin (e.g., "drug dealer"), but 
more positive results for George Zimmerman (e.g., "hero"). 15 

In scope for algorithmic unfairness. The system processed and showed offensive content from 
users, in a way that could be seen to reinforce existing social biases regarding race. 

Google Search Results for Black Girls & Pornography 

In 2013, a researcher expressed concern that search queries on Google for "black girls" historically 
yielded a high percentage of pornography. 16 

In scope for algorithmic unfairness. The system showed results that are offensive and reinforce 
existing social biases regarding race. 

Google Image Search Results for Physicists 

Google image search results for "physicist" show predominantly men. In reality, roughly 20% of 
physicists are women. First, imagine that the image search results show 1% women. Second, 
imagine instead that the image search results show 20% women. 

In scope for algorithmic unfairness. In the first case, when the image search results show only 1 % 
when the reality is 20%, the system is amplifying an existing bias in society. In the second case, when 
the image search results show a percentage similar to the current reality, the system is reflecting an 


14 Giles Hogben, Alex McPhillips, Vinay Goel, and Allison Woodruff. Allegations of Algorithmic Bias; Investigation and Mela-Analvsis . 
September 2016. ao/alleqations-of-alqorithmic-bias 

15 Safiya Umoja Noble. Trayvon, Race. Media and the Politics of Spectacle. The Black Scholar. 44(1 ):1 2-29, Spring 2014. 

16 Safiya Umoja Noble. Google Search: Hvper-visibilitv as a Means of Rendering Black Women and Girls Invisible . Invisible Culture: 
Issue 1 9, October 29. 2013. 
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existing bias in society. Both cases fall in scope for algorithmic unfairness because they reinforce a 
stereotype about the role of women in a scientific discipline. However, while both are in scope for 
algorithmic unfairness, remediation may be more likely in the former case. (As with the other cases, 
whether and how to remediate would fall to product policy.) 

Microsoft Kinect 

News media in 2010 and 2017 reported on developers and users of Microsoft's Kinect experiencing 
difficulties with facial recognition or motion detection for users with dark skin in low light conditions. 
Affected users could not use certain system functions including automatic sign in to the user's 
profile,' 7 or use software that required motion detection to function.' 8 

In scope for algorithmic unfairness. The predictive system performed poorly for users with dark skin, 
which is associated with the sensitive characteristic race. Users considering purchasing the Kinect 
or software that requires the Kinect would be unlikely to suspect that their skin color might make it 
difficult or impossible to use advertised features (such as automatic login using facial recognition, 
gestural menu interactions, motion controlled gaming). Gaming hardware and software purchases 
are commonly non-refundable after being opened, so a user who realized the Kinect or its software 
could not track them would be left with a non-functional product they could not return. 

Survivalist Game on the Play Store 

In January of 2016, a series of Twitter posts and a Change.org Petition raised concerns about a 
Survivalist game on the Play Store, claiming it gamified killing aborigines. 

Not in scope for algorithmic unfairness. There were concerns that the content was offensive, but an 
algorithm was not involved. 

Airbnb Acceptance Rates 

Researchers from the Harvard Business School conducted an experiment on Airbnb and found that 
applications from guests with distinctively African-American names were less likely to be accepted 
relative to identical guests with distinctively white names. The study could not identify whether the 
unfairness was based on race, socioeconomic status, or some other factor.' 8 

Not in scope for algorithmic unfairness. The unfairness was on the part of individual users ("sellers" 
on Airbnb) and there was no algorithmic component to the unfairness. While this is out of scope for 
algorithmic unfairness, Airbnb did however recognize that their design choices unrelated to 
algorithms could be facilitating unfair behavior. Accordingly, it pursued non-algorithmic 
remediations, such as modifying its user interface design and its end user policy. 20 


17 Brendan Sinclair. Kinect has problems recognizing dark-skinned users? Gamespot, 2010. 

18 Andy Trowers. How We Accidentally Made a Racist Videogame . Kotaku, 2017. 

18 Benjamin Edelman, Michael Luca, and Dan Svirsky. Racial Discrimination in the Sharing Economy: Evidence from a Field 
Experiment . Harvard Business School NOM Unit Working Paper (16-069), 201 5. 

20 Laura W. Murphy. Airbnb's Work to Fight Discrimination and Build Inclusion: A Report Submitted to Airbnb . September 201 6. 
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FAQ 

Is personalization the same as algorithmic unfairness? 

Generally, no. Personalization is algorithmic behavior that presents different results to different 
users. Personalization is very often beneficial to users and is not necessarily unfair. In fact, 
personalization may sometimes remediate unfairness by identifying an individual's interests more 
precisely than would the use of a sensitive characteristic. 21 However, in some cases personalization 
may be strongly associated with a sensitive characteristic in a way that causes significant harm to a 
user or users, in which case it would be algorithmic unfairness. 

Can any characteristic be sensitive? 

No. For example, "people who like yellow" or "people with pets" are unlikely to be sensitive. 22 

Which characteristics are sensitive? 

Sensitive characteristics may be determined by legal considerations or by more broad principles. 
Legally determined characteristics may vary by jurisdiction, sector, or other factors. Sensitive 
characteristics determined by more broad principles are especially likely to include characteristics 
that are associated with less privileged or marginalized populations (particularly when such 
characteristics are immutable), may be socially undesirable, or may be associated with civil liberties. 

Specific examples may include race/ethnic origin, gender identity, sexual orientation, religion, 
political party, disability, age, nationality, veteran status, socioeconomic status (including caste and 
homelessness), and immigrant status (including refugee and asylum seeking status). Further, there 
may be emergent sensitive characteristics for which we do not yet have a name, or which we have 
not yet anticipated. 23 

Are proxies for sensitive characteristics covered? 

Yes. Proxies are characteristics that are highly correlated with a sensitive characteristic. Actions 
based on close proxies for sensitive characteristics are in scope for further investigation. For 
example, if interest in hip-hop music is highly correlated with being Black, targeting users who like 
hip hop music may result in algorithmic unfairness based on race, regardless of intent. 

If a system's behavior is caused by societal bias, can it still be algorithmic unfairness? 

Yes. Societal bias that is reflected in algorithmic behavior is a central issue in the external public, 
media, and regulatory concerns about algorithmic unfairness. 24 ' 2526 ' 27 Additionally, one often doesn't 


21 James C. Cooper. Separation and Pooling . George Mason Law & Economics Research Paper No. 15-32, March 2016. 

22 httD://civilriohts.findlaw.com/civil-riohts-overview/what-is-discrimination.html 

23 danah boyd, Karen Levy & Alice Marwick. The Networked Nature of Algorithmic Discrimination . In Seeta Pena Gangadharan, 
Virginia Eubanks, and Solon Barocas (eds), Data and Discrimination: Collected Essays. Washington, D.C.: Open Technology Institute, 
New America Foundation, pp. 53-57,2014. 

24 Tarleton Gillespie. The Relevance of Algorithms . In Media Technologies: Essays on Communication, Materiality, and Society, 
edited by T, Gillespie. P. Boczkowski, and K. Foot. MIT Press, Cambridge, MA, 2012. 

25 Solon Barocas and Andrew D. Selbst. Big Data's Disparate Impact . 104 California Law Review 671 (2016). 

26 Executive Office of the President. Big Data: Seizing Opportunities. Preserving Values . May 2014 
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know the root cause of algorithmic unfairness without an investigation, so any real-world process 
(e g., incident response) is best served by encompassing a wide range of potential causes. 
Accordingly, this definition covers a wide range of sources, from machine learning classification 
errors to societal bias. However, the nature of the root cause may affect product policy's position on 
whether or how a given algorithmic behavior should be addressed. 

If a representation is factually accurate, can it still be algorithmic unfairness? 

Yes. For example, imagine that a Google image query for "CEOs" shows predominantly men. Even if 
it were a factually accurate representation of the world, it would be algorithmic unfairness because it 
would reinforce a stereotype about the role of women in leadership positions. However, factual 
accuracy may affect product policy's position on whether or how it should be addressed. In some 
cases, it may be appropriate to take no action if the system accurately affects current reality, while in 
other cases it may be desirable to consider how we might help society reach a more fair and 
equitable state, via either product intervention or broader corporate social responsibility efforts. 

If a system's behavior is not intended, can it still be algorithmic unfairness? 

Yes. If the behavior is unfair, it meets the definition regardless of the root cause. 

If unfairness is executed by an algorithm but is the result of a human decision, can it still be 
algorithmic unfairness? 

Yes. For example, if a human chooses unfair keywords for ads and those choices result in unfair 
algorithmic choices of what ads to show to a user or users, that would fall within the scope of 
algorithmic unfairness. 

Does this definition include the use of data by multiple parties? 

Yes. For instance, decisions which may result in data being provided to third parties via APIs, being 
sold to third parties, or acquired via federated identity should consider the possibility that that data 
could be used to power unfair algorithmic decision-making. 

What is the relationship between this definition of algorithmic unfairness and the fairness measure 
in Hardtetal. (2016)? 

The fairness measure in Hardt et al. (2016) 28 is a statistical guarantee that, in general, members of 
one category are classified with the same accuracy as members of another category. For example, if 
Black people who apply for credit cards are classified with lower accuracy than white people (e g., 
Black people who would actually repay their loans are wrongly classified as bad credit risks), the 
fairness measure in Hardt et al. (2016) would find that to be a problem. However, if most people who 
happen to have a particular sensitive characteristic are correctly classified as being very likely to 
default on their credit cards (regardless of whether that classification is done based on that 
sensitive characteristic or based on some other associated variable), the Hardt et al. (2016) measure 
would not detect it as an issue. Therefore, the fairness measure in Hardt et al. (2016) statistically 
tests for a certain type of machine learning classification error that would constitute a specific type 
of algorithmic unfairness, encompassing some but not all of the cases covered by the current 
definition. 


27 ACM US Public Policy Council. Statement on Algorithmic Transparency and Accountability . January 12,2017. 

28 Moritz Hardt, Eric Price, and Nathan Srebro. Equality of Opportunity in Supervised Learning . arXiv.org, October 2016. 
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Revamping News Corpus 


Goal: Establish "single point of troth" for definition of "news' across Google products. Mitigate risk of 
low-quality sources and misinformation in Google News corpus. 

Status: Define new Google News corpus utilizing existing infrastructure and tools while integrating 
different quality tiers and labels, algo-human content review, new cross-product review/exclusion 
pipeline and updated inclusion Ul. 


■plate (WIP) / Stress Test publisher list 
NEXT STEPS: Research and gather info about possible signals to leverage for product teams to make 
source quality decisions. Finalize and test new rater inclusion guidelines with standards to combat 
misinformation. 


Trust Project & Nutrition Labels 

Goal: Develop transparency standards that help people easily assess quality and credibility of 
journalism. Work across News. Nutrition Labels, and Search Console / 3P structured data teams to 
incorporate Trust Project data into future product plans. 

Status: Launched eight trust indicators with a dozen publishers going live with schema 
implementations. Launched publisher Knowledge Panels in Search. 

contactsHMM 

MORE INFO^ublisnemWauncn^Snouncement^ne Trost Project 

NEXT STEPS: Complete PRD Addendum and SD Playbook for markup support in Search Console. 


Reviewing Rater Quality 

Goal: Evaluate general performance of Ewok rater evaluations in select countries for News corpus inclusion to 
uncover potential red flags or unusual patterns. 


httpsi//groups-googte. co m a google. com/fo rurn/#?msggoogte-90S-engVJFtQ21Cww4/01-1bXkBQAJ 
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Status: Analyze low-quality and high-quality rater comments, examine inconsistencies and rater 
inclusion history. 

contactsHI^HH 

MORE INFO^/^Pnvilege^n^Confidential 


PRODUCT OPERATIONS 


Project Purple Rain: Crisis Response & Escalation 

Goal: Establish and streamline news escalation processes to detect and handle misinformation across 
products during crises. Install 24/7 team of trained analysts ready to make policy calls and take 
actions across news surfaces including News. News 360 and Feed. 

Status: SOS Alerts, Crisis Response. HotEvent. and T&S Incident Management teams are 
collaborating to identify a narrow set of queries that would be used to manually trigger flight-to-quality 
in Search. T&S Incident Management team is currently looking to expand and share resources with 
teams that currently handle Suggest and WebAnswers escalations. 

contactsHHHHHHHHH 

MORE 


Policy Development & Enforcement 

Goal: Develop and proactively enforce misformation policy that scales across News products. Explore 
cross-product toolset for removal enforcement. 

Status: Drafted policy for Coordinated Inauthentic Cross-Border Information Operations (CICBIO). 
Reviewed proposed policy with Search & News leadership. 

CONTACTS 

MORE INFO^re^proposan^xpan^^nc^nrorcemen^ 

NEXT STEPS: Set up workflows and tools with TAG team and other stakeholders. jeffreyc@ and 
@jacobhelberg@ to scope "badness baseline rating" and other tools for top of PO escalation funnel. 


ESCALATIONS 


Italian sites spread misinformation: PR escalation following Buzzfeed article. 3 sites rejected 
following ewok and manual evaluations. 

Hoax science stories in News: Junk science articles flagged by internal Googlers and written up in 
blog. Status pending: 5 sites submitted for ewok re-evaluation. 3 rejected 

Dylann Roof sentencing resurfaces in News: Danny Sullivan's sleuthing uncovered cause as 
publisher error that spawned copycats. (Resolved on Twitter) 

CA shooting rampage: No major issues across products. PR post mortem with open questions and 
follow-up here. 

TX church shooting: We surfaced some misleading tweets in Search and some low-quality videos in 
YouTube which resulted in a negative press cycle. Danny Sullivan responded with a senes of tweets 
(PR summary) 


https :Z/groups.googte.conVai'google.com/Torijm/#’msgi , google-sos-eng/1(JFtQ21Cww4/01-1bXI(BQAJ 












Expanding Collaboration 
for News Quality 

Fatih Ozkosemen, Trust & Safety Search 
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Google efforts to address 
Fake News 

What Google is doing to address the problem 

ao/FakeNews 

Gustavo Fuentes-Rivera, Pauline Peyronnet, Mark Lyall 
Dec. 2016 
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Google 



o 

Users have access to a large variety of news 
sources 


Google 



The web offers a large 
variety of options. 

From reputable, traditional 
news sites to newer, 
relatively unknown news 
outlets. 
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Many users access news sites via FB News Feed 
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but some bad actors came into play 
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TRENDING 0) A © 

BREAKING: Fox News Exposes Traitor Mcgyn 

^ Moflyn Kelly 

6IK people talking about this 

Kelly, Kicks Her Out For Backing Hillary 

** Anthony Weiner 

Megyn Kelly has brought a lot of heat on her many endeavors to btemoh 

13K people talking about this 

GOP presidential candidate Donald Trump, and me Fox News viewers are 

/v» Beyonce 

not satisfied The Kietfy File's ratings have even faflen behind Maddow 

41K people talking about thla 

Show on MSNB. whtch is ftsgraoeful VIA Natcnal insider Pollies Via 

** Governor of North Dakota 

Conservati 

4.7K people talking about this 

ond ngtho’od com 

rS Don Cheedle 


ItK poopio taSung about this 

Posts by Gregory M. Lee. Chrta Conerd and 13 hours ago 

^ Mylan 

others 

2S0K people taking about this 


In the context of the US elections, a number of bad actors used the 
Facebook News Feed to promote fake news sites 


Google 


And the issue gained notoriety 
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Tech fe Chart of the Day 

Fake News Is A Real Problem 

Facebook engagement of the top five fake election stories' 


HMdlrw 

"Pope Francis Shocks World. Endorses Donald, 
Trump for President. Releases Statement" I 

IrxWnt ttw led ' 

“Wikileaks CONFIRMS Hillary Sold Weapons to 
ISIS...Then Drops Another BOMBSHELL' Breaking News" 

The invder 1 


960,000 


“IT'S OVER: Hillary's ISIS Email Just Leaked & It's 
Worse Than Anyone Could Have Imagined" 

lndtn*th* 



"Just Read The Law: Hillary Is Disqualified From 
Holding Any Federal Office" 

fntiing ttw r«d 



"FBI Agent Suspected in Hillary Email Leaks Found 
Dead in Appartmcnt Murder-Suicide* 


567,000 


Total Facebook engagement for top 20 election stories (August-election day) 

Fake news 8 7 m 

Mainstream news 7J m 


Business Insider 
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We discovered that some of these sites were 
using AdSense to monetize their traffic 


USA TODAY 


IPOOU UM MONfV n<H 


.USATODAY 



Google to pull ads from fake news sites 


SAN FRANCISCO - Aim) crtttctwn that Uko now* 
•UM hove contributed to mwntormrt to n and 
increasing poton/oOon among the American public 
Google announced Monday night that it wtl stop 
allowing toko newt trtea to uee its ad toftware 

Faoobook tot towed with a tinnier pokey 'In 
accordance with the Audience Network Pokey, we do 
not Integrate or display ads in apps or sites 
containing content that is illegal, mis leading or 



and we took quick action, 
updating our AdSense 
policies to address 
'Misrepresentative content' 
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o 

With this update we disallow sites that mislead 
users 


Approved Public Statement: WeVe updated our publisher policies and will start prohibiting Google ads from 
being placed on misrepresentative content, just as we disallow misrepresentation in our ads policies . Moving 
forward, we will restrict ad serving on pages that misrepresent, misstate, or conceal information about the 
publisher, the publisher's content, or the primary purpose of the web property. 


This update solved the problem of fake news sites monetizing with AdSense. 
We're working on a similar update for AdX to cover all our bases. 

Additionally there is a working group established to assess whether changes 
are needed for our platform products (DBM, DFP) 
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But then another problem surfaced: Ads 
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SHOCKS Fans! 
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There's No Joy For The View 



Joy Baher shocks her fans and 
you wont believe why! 
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Google 



How are we addressing this? 


We received escalations 
from publishers about ads 
promoting misleading 
information/sites. 


Internal Only 

Proprietary * Confidential 

o 

Global Partnerships 


(i) Reviewing suspect ads and taking action in cases where 
there is a violation of our ads policies. 

(ii) Assessing what improvements can be done in automation 
support and/or review procedures 

(iii) Preparing guidance for publishers on what steps can they 
take on AdX to block ads that they consider undesirable 


Google 


Internal Only 

Piopriatary« Confident*! 








Resources for external communications 


o 


"Misrepresentative Content" update on AdSense Policy - Comms Doc 


Approved Public Statement: We've updated our publisher policies and will start prohibiting Google ads from 
being placed on misrepresentative content, just as we disallow misrepresentation in our ads policies . Moving 
forward, we will restrict ad serving on pages that misrepresent, misstate, or conceal information about the 
publisher, the publisher's content, or the primary purpose of the web property. 


AdSense Program Policies - Prohibited content 


Google 


Misrepresentative content 

Whafa the policy? 

Users don't want to be misled by the content they engage with online. For this reason, Google ads 
may not be placed on pages that misrepresent, misstate, or conceal Information about you. your 
content or the primary purpose of your web property. 

Examples v 


Internal Only 
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Additional resources (internal) 


Internal FAQ for AdSense Misrepresentative Content Policy 


FAQ on "Fake News" 

Q: What is Google's policy on "fake news"? 

A: Google does not have a specific policy related to "fake news". "Fake news" is not a clearly 
defined concept and any policy related to it would require Google to make the assessment as to 
what is the truth around the world. Google does have key policies that protect users from 
misrepresentative content and hate speech which are addressing the badness being seen on 
the network. 


Internal Only 

Proprwtwy * Confldmtal 
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Summary 


Two key areas of concern: 

1. Monetization 

What is Google doing: 

We have updated our AdSense policies to restrict ads on sites that 
misrepresent, misstate, or conceal information about the publisher. 
To cover all bases we will make a similar update to AdX policies. 


2. Ads 


What is Google doing: 

> We are reviewing suspect ads and taking action in cases where there is a 
violation of our ads policies. 

> We are assessing what improvements can be done in automation 
support and/or review procedures 

> We are preparing guidance for publishers on what steps can they take on 
AdX to block ads they consider undesirable 


Google 


Beyond 


Increasing policy coverage over 

• Abuse 

Other spam types, malware, deception etc. 



• Fringe/controversial 

Factually incorrect, fake, irrelevant, non-news etc. 


• Sensitive 

Hate, geo-politically sensitive, diversity & bias etc. 


• Inappropriate 

Sexually explicit, graphic violence and vulgarity etc. 



T&S 
can help 


Proposed 

Als 


• Badness signal discovery & data sharing for policy 
enforcement 

• Multi-lang market analyst support across all top 
languages globally 

• Leveraging TVC resources where automation is not 
possible 

• Aligning with policy (GPP is part of T&S) 

• T&S Eng resources 


• setting up a metric for corpus quality 
News corpus badness rate 

• broader policy enforcement and abuse detection 

Beyond webspam 

• better detecting low quality 

lq sources + Iq content in hq sources 

• clean & regularly sanitized news corpus 
both for site registration + ongoing 
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Fairness 


Fair is not the default 

go/'air not default iQYejoyS' ; Google Confidential 











































It's the result of the photographer being 
average height. From an outlier’s perspective, 
the world might look significantly less uniform. 









This is a talk about 
the role of humans in 
machine learning. 





But really it’s a talk about 
the role of humans in 
decision making. 




"fit's true that they can follow instructions at superhuman speed, with 
superhuman fidelity ana over unimaginable quantities of data. But these 
instructions don t com a from nowhere. Although neural networks might 
be said to write their own programs, they do so to words qoais set by 
humans, using data, collected hr human purposes, if the data is skewed, 
even by accident the computers will amplify injustice. " 


— The Guard .in 














%'s true that they con follow instructions ot superhuman speed, with 
superhuman fidelity and over unimaginable quantities of data. But these 
instructions don't come from nowhere. Although notsfal networks'might 
be said to v/rifh th&Y own programs, they do so towards goats set by 
humans, using data collected for human purpo^nrine OGTtrisskewed 

(amplify injustice ) 


• — The Guardian 









Recidivism 
software is 

biased against 
black people 


Photo-editing 
app makes faces 

look more 

Caucasian 


Researchers 
claim facial 

attributes predict 

criminality 


Machine Dios (PrePubiica} 


FaceArJP apcieqizcs forbu>loinq a Phyjiegnerrv 's Ncw Clc-the l. 

racist Al iTochCrunch: iMediuTij 





When we presume that h jman judgment can—or 
should—be removed from the loop, the result is an 

unconscious bias network effect. 


And we (Googlers) are just as susceptible to this effect os our users. 







Training data are 
collected and 
classified 


Algorithms are 
programmed 






Mecha are 
filtered, ranked, 
aggregated, or 
generated 


Algorithms are 
programmed 














Training data are 
collected and 

classified 








Media are 
filtered, ranked, 
aggregated* or 
generated 


Algorithms are 
programmed 














Training data are 
collected and 

classified 


People(f/ke us) 
are programmed 
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Unconscious bias gets reinforced in the training cata 
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Training data are 
collected and 

classified 
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Unconscious bias affects the way we collect and classify data, design, and write code 








If we ere going to make the world T s information 
universally accessible and useful, we must stnve to make 
the products we are developing—apps* infrastructure* 
models* and more— work for everyone. 


Learn more *u ou n 1 ■ ■ - 



If we are going to make the world’s information 
universally accessible and useful, we must strive to make 



the prod ucts we are d evelopjp^^apps7mt restrict u re 
models, and more^-work for everyone. 


t 



Learn mortal ■ io ni -'aiiiess 




And unfortunately, we (humans) have a 
history of making product design 
decisions that are in line with defaults, 
and nof with the needs of everyone. 


■f-oi me his talk flue teim "de&ulT feuae?) <o represem eh* imp a aclihjtfes. a culture cr locipy 

flOi'ai Ally £ubs«'it.c: 10. lyeico -i v/'irkwti&iscloiis Morecwt. Waio :op' io'-aw-JJ_ 'j_ 





Female drivers are &7% more likely to be severely 
injured in an auto accident because, until 2011, 
female body-type crash test dummies weren't 
required by the U,S, Department of Transportation, 


out 






Until the 1990 T s, virtually all color calibration *'or color film 
was based on Caucasian skin tones* It wasn’t until Kodak 
started receiving complaints from chocolatiers and wood 
furniture manufacturers that they began to invest research 
into supporting a broader range of brown tones. 
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Sn the top 100 grossing US livo-action 
films from 2014-2016* male characters 
were seen and heard nearly twice as 
often as female characters. 


SOUttE 
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Where are these 
disconnects coining from? 





Are automakers 
ntentionally trying to 
harm women? 


Was the decision to 
color balance for light 
skin overtly racist? 


Are creatives in the film 
industry marginalizing 
women on purpose? 



These are often the behaviors of rational 
actors making what seemed like 'obvious' 
choices, without malice or ill-will. 






Take for example the 
case of “David” 











Or was it "Diana”? 







For the purposes of this study, it was both. 


Participants watched a series of videos of babies. 
For each video, half of the participants were told 
a boy's name, the other half a girl's name. 


.tutus 

Slj'Jv pr i ^ FyHL.ri.HH 3elv; dei. C 
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When the babies did something 
unambiguous, reports were not 
affected by the perceived gender. 

If the baby clearty smiled, for example, everyone 
said the baby was smiling or happy, 


3H1IU5 


Qiiyeixft, A 5 k civ or ue E,"r y 11, Beholdr. Condi v i >c c-yn 
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Then the babies played with a 
jack-in-the-box toy. When itsuddenly 
popped up t the child was startled 
and jumped backward. 


MUttlfc 
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And most of us would agree that 
it would be irrational behavior to 
treat a fearful child the same way 
we’d treat an angry child. 



m tt knowledge of a avid's gender affects adiMs" perception of 
that child, then male and female children ace going to elicit 
different reactions from the world, different patterns of 
encouragement These perceptions matter, even in parents 
who ore commuted to treating sons and daughters alike. “ 


- Elizabeth Spclke 


EQ-BCES 
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Same child, same reaction 
different perception. 





Human perception drives 
virtually every facet of 
machine learning. 



I propose we* make machine-learning 
intentionally human-centered and 
intervene for fairness. 


■ = Humans nils Isn't s*methiiq any company si-ionid be-iwrvg in isolation, but we're in a gotxi posltorito start 






Tenets 


Designing for fairness 




Be 

Accountable 


We can’t take our hands 

off the steering wheel. 

In rejecting the myth of neutral data, 
we are committing to be more 
conscious and! conscientious. 



Be 

Accountable 


Present day 
U Robots 
>* Yada yada y a d a 
u F utur e 

Present day 

'> Humans 
*> Still hurra ns 





Be 

Skeptical 


Challenge assumptions at 
every turn. 

We cant blindly rely on the systems 
that underpin conventional wisdom. 



D2 

Be 

Skeptical 


Journalism 

Fake news is. indislingi.isha.hlG 

Customer reviews 

Vink: reviews ^knvv riyntricjn sir, or 

Standardized tests 

SAT scoies don't predict oracles 


Medical science 

EnoaniriFints o jr r-rscrui r 

Crime statistics 


gfl^kHifjatiiiijsJa_L£a.i 




Be 

Humane 


Success metrics should 
bring out the best In 
human nature. 

Standard engagement metrics confine 
people to whatever they’ve done 
before, rather than empowering what 
they’re capable of doing next. 
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B6 

Humane 


Time well spent 

ti m owoll spc ntio 

Learning and expression 
Exploration and connection 
User-defined goals 



Be 

Humble 


We don’t always know 
better. 

The tech Industry is in love with 
“disruption”, but frequently that 
means imposing a vision of the 
future onto users and expecting 
them to adapt. 



Be 

Humble 


“There are two ways to get people 
used to automation. The soft and 
fuzzy way, ...just keep reassuring 
people until they re comfortable. 
And then there's the second way: 
let the humors fa/ce control when 
they need to. ” 


-ms 





Augmenting - 
Supporting 





An 

In practice 

ML user journey in 12 phases 




We come up with the ideas 

What will the model actually predict? 
Who stands to benefit the most from It? 







We conduct the market research 

How dc users describe the benefits of ML? 
What reference points are they bringing? 




We design the experiences 

How are people solving these problems today? 
How might ML Improve things? 












PHASF 06 


We design the Rater protocols 


Hypothetically speaking, would this oe an 

unambiguous task for end-users to perform? 


{A urflor s .nole: r.hs use of Raters is primarily fbr supervised earning, labeled data—ground truth ar otherwise—have a wide variety 

otQfjpi'taUcri. so cfrosc Jo iV'CJuoc fftfs os o fyominciH jotoso. ! 7c,'rTi'o, r 'cc , rnf*oC rcorflr'no i's SikoSy (ft ooiY oi'oco tt.’ovVo entirely absent 




Sidebar on Raters 


Speed and Agreement nre the bedrock measures of H c ck workers". 
The general aea is tfiat if a lot of people cnn perform many quick 
tasks, fhe sheer volume o ; consensus will oalance their lack of 
Individual expertise. But without careful consideration for the 
diversity of Raters, chck-wprk turns into exponential crcupthink; 
baking cultural nlas.es directly into training data. 







We try to label all the things 

Where did you acquire the knowledge 
necessary to successfully label these data? 






We recruit and test with users 


How will the backgrounds and life experiences 
of your testers Influence your doctors? 




we interpret the user feedback 

What decisions fire you trying to make? 
What evaluation metric w II you prioritize? 






Finally, I’d like to offer 3 human-centered 
diagnostics when designing with ML. 

If you’re finding it tricky to answer these, it night be a 
signal to slow down and take a closer look. 



01 

If a human were to 
perform this task, what 
would ‘appropriate’ social 
behavior look like? 


What interpersonal cues might be relevant 
that are missing from your input or trterface? 

E.g. body language, tone of voice. 


Tfske sulocomptete for example: In what context would 
it tie acceptable to finish another person’s sentence 
before they've stopped talking? 

Bear in mind that no one culture is capable of 

ropros anting universal norms, ospocially for social 

interactions, so we need to he mindful of our intuitions. 
The psychological offocts of algorithmic discii mi nation 

Mkelv nrirrorthggr* of social discri mi nation . 


autocomplete is an 

4 a 

autooompiete is an interruption 
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02 

If a human were to 
perform this task, might 
we call it an expression of 
their personality? 

Tasks that have unambiguous utility, are 
perceived as repetitive or boring by users, 

arncttor heneFrt from super fast response times 
are ideal candidates for ML augmentation. 


Reply to all 


Greal idea! 


I like that kJie-y. 


Please I ike rTitj. 


Grey area: Suggesting a reply in an email or SMS 

likely has a priming effect; impacting the user's 
response even if they don't use it. And the fact that 
suggestions are even offered may lead the recipient 
to question the authenticity of the sentiment. 

Augmentation: The goals Of o self-driving c?r are 
unarmblguous, and the benefits- of a computer's 
superhuman reaction time offer objective utility. 

No on? (hopefully} w?uld 5oy o driver got Into an 
accident because they were expressing themselves. 
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Who are you? 

... OK, now what do your 
data teach you about 

everyone else ? 


Our traits don’t necessarily define us. but it’s foolish to 
pretend we don’t see them. By taking the potentially 
uncomfortable step of inventorying these traits— 
physical, social, cognitive?, and olharwiscr—wo Vo 

getting proximate to those who are reminded! of their 
differentnes s every time a 'default 1 is invoked in 
day-to-day life. 


- ism., 


Whte 
A parent 
Visually impaired 
Agnostic 
Insured 

Urbar>-tfwel ling 
In my SO'-f; 

Cisgendtr 

99% percentile height 


Male 

Affluenr 

Physicaly capable 

cuitLii Aliy Jewish 

A homeowner 

No? college educated 

Married 

Heterosexual 

In good menial Ivealiti 


A speaker of “starvdbrcT U.5. engilsb 
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Because hopefully that 
can help your world look 

a bit less like this... 
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Because hopefully that 
can help your world look 

a bit less like this... 

...and 0 lot more like this 









Thank you 


Dive deeper at ao/ml-fairness 




Appendix 




The goal of this section is to examine the 
disconnect hetween our expressed moral code 
and the unconscious biases that frame our 
perception and subsequent decisions* 



oi Gender roles 
in film 




CENDIBHOMS IN FILM 


In 2015, women shared top billing with men in the 
top four grossing live-action films in the U-S- 


How much were female characters 
seen and heard compared to men In 
those films? 









|j| hlllHRIII I •: iN IILM 


In 2015, women shared lop billing with men in the 
top four grossing live-action film;: in the U.£. 


female characters 
seen and heard compared to men 

those films? 


SOUSCf 
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GENDER ROLES IN FILM 


The numbers Improved slightly for 
the top grossing films featuring 
female leads. 

But still contain some surprisingly 
disproportionate numbers. 










The numbers Improved slightly for 
the top grossing films featuring 

rernaie leads 

But still contain some surprising!'/ 
disproportionate numbers. 


a?cjuiJa'/l5..l3sULuLB on bcndc-'in -Madia 







t£N DEG BOLES IN FllH 


But the gap increased substantially 
for the top grossing films featuring 
male leads. 
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But the gap increased substantially 
for the top grossing films featuring 

male leads 


HIKE 

tiPCTM Uvwir, Hatrtulc C'.i 'j'srtlgrlri.MgdiD 



87% 

38 % 














GEM DOLES INI FILM 

Overall, for the top 100 grossing 
live-action films from 2015 in the U>S. 

Ha lt* ci aracie rs mere see n Malectia v< cleft we re hea rt 

1.84x 1.78x 


k more than female character 


a more ttan female characters 







GE NDI l ROlti IN l ILU 


Related research 


’A 1 ha n w cm Bn a nd m an ; - e a [ -„h e- e’< set turn e 

amount, women are peceivBj to bn speaking 

22 % 

more than men 


When in a mixed-gender group, women 

speak I esi than in m u ntii th ey cornpri sg 

80 % 

of the group 


SOURCES 

breaker sex aid □■r^K.^jjDrcrtcfnrncntcrf lalk.Cuberand Sontt 

.^■ "'lr lni--ii,- ilv !" ■■ln>ar:-|-..T P.-riirinnl cn. Kmnr///:7Hnrlr-CTr. n>- Shiner 

JchndfeiaJitt’nea-bnBiawTCiit o1 Yncmc-rcn inclan HK-.J'jt/.artl j'.v Ls»j:c.i 


In m jl* mg the to p 2k 0 gro ^ t i rg f i I me 

c f 2 D 15 in the U.S.. worn Enfield 

19 % 

cf behind - the--scene s creative role i 



One more observation 



Haw you ever noticed the “gender* 
makeup of Sesame Street? 






All of the original starring monsters 
arc regarded with male pronouns, 
despite having no d seernable 
physically gendered traits. 




It wasn't until 1993—24 years after 
tho show started— that a fcrmalo 
monster named “Zoe" was added to 
the core cast of characters. 


I 




02 Skin tone in 
photography 




SKIN TONE IN PHQTQGGAPHV 


This is a ^Shirley Card” 


Named after a Kodak studio model named Shirley 
Page, they were the primary method for 
calibrating color when processing film. 
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S.KIN TONE IFJ PHQTQGRAPlHV 


Until about 1990, virtually all Shirley 
Cards featured Caucasian women. 
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SKIN TONE IN PHOTDGRAPtfY 


As a result, photos featuring people 
with light skin looked fairly accurate. 
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SKIW T3ME IMPHOlIOLItAPHV 


Photos featuring people with darker 
skin, not so much... 


SHREK 
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SKIM TONE IHFHMCDAPHY 


And when there was a mix, the 
difference was most noticeable. 


Hums 
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SKIMTQNE IN PHQTQIfiAPH'i 


As society became more Integrated, 
photographers found workarounds. 

The most common techniques were to shoot with 
significantly brighter lights (making the room 
really hot!), using a stronger flash (42% brighter!},, 
and prpnarinrj separate ram eras with different 
calibrations for people with different skin tones. 


sums 
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: KIN TONE IN PHQTDGfiAPHV 


But motivation for innovation came 
from chocolatiers and wood furniture 
manufacturers. 

Kodak was receiving complaints that they weren't 
getting tine right brown tones on chocolates, and 
that stains and wood grains were not true to life. 


MIKES 
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5MN TONE IN PHOTOGRAPHY 


Related research 

As or ?0R across all 2 mi Hi m Imp iriUUsociation As oi 2014, across all iJ S white IAT 

Test 0A1) participants. participants.the median"Dscore" was 

51 % 0.402 

showed a moderate to strong bias forwhite faces indicat ng a moderate bras forwhite faces 
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As of 2011, of adults workirg in Erlginear!ng 
rales in the U S. fage 16 and over. 1 

4 . 8 % 

were black 



03 Automotive 
r ~ safety 




AOTQHOTIVE SAFETY 


Until 2011, female body-type crash 
test dummies were not required by 
the United States Department of 
Transportation. 
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AUTOMOTIVE. SAFETY 


As a result, female drivers are at a 
higher risk behind the wheel. 

Odds a female dri m will sustain severe i njuri os in an accident 

47 % 

4 higher than a inale driver 
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AUTOMOTIVE SAFIN 


Things are improving, but the target percentile for 
female test dummies remains problematic. 

M ale body p ercemiI c Fern ale body pBrcenii I e 

50th 5th 

5*9*116 lbs 411’lOBIbS 


sum 
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automotive s^fetv 


Related research 



Scenarios t#sied usingfemale liarlv-lypc As of 2 015, l h & rm mh&r of wo me n working in the At of 2115 r the number tr tar buying 

crash test d imrnies in Ihe d river's seat m otor ve hide man ufacturi ng ind ustry decisions i nil uen red by women 


1 of 3 26,7% 
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We can’t remove human 
perception from the loop. 


And we can’t be gripped by 
inaction either. 




The inequity demonstrated in these examples may feel 
overwhelming, perhaps even a little disheartening. 

But we’re in the right place at the right time and in 
the right industry to do something about it. 



